Add capability reliability ledger (#186) - #326

Source Issue

Closes #186
Parent Epic: #184

Agent Run

Runtime: claude-code
Run ID: n/a

Summary

This PR now contains both the architecture and the full implementation of #186 — Add capability reliability ledger, per docs/architecture/issue-186-capability-reliability-ledger.md and ADR 0012.

Two new immutable tables — capability_attempts and capability_attempt_adjudications — record comparable evidence for how reliably a specific capability has performed in a specific project/scope/runtime/policy cohort, without collapsing materially different work into one score and without letting a worker grade its own output.

Contracts (lib/reliability/contracts.ts): capability-key grammar (workpackage:<role>/<capability> or operation:<id>@<version>), domain-separated SHA-256 cohort/scope/runtime/policy fingerprints matching the ADR 0011 construction, closed verification_mode enum, and the ReliabilitySummary shape.

Metrics (lib/reliability/metrics.ts): computeReliability is a pure function — no clock, no I/O. Below a minimum sample size it reports insufficient_evidence with null rates; if any in-window attempt's linked outcome has changed since ingest it reports evidence_drift and withholds all rates; critical failures are always counted regardless of state. No materialized cache in v1 — deliberately, see §6.3 of the design doc.

Persistence (db/migrations/0031_capability_reliability_ledger.sql): both tables are append-only via BEFORE UPDATE/DELETE reject triggers; adjudications enforce gapless per-attempt sequencing via a BEFORE INSERT guard; every text column is closed by a CHECK — there is no free-text column anywhere in the ledger, so a model's prose, a file path, or a secret cannot enter it even by accident.

Ingest (worker/reliability/ledger.ts + context.ts): idempotent on (execution_outcome_id, capability_key), wired best-effort into the three existing work-package outcome boundaries (admission block, completion, failure), the ADR 0011 operation-finalize path (after its transaction commits, never inside it), and review-gate decisions as append-only adjudications. independent_agent verification is refused at ingest since #188 has no producer for it yet — the honest current state (completed, but nobody independent checked) is reported as an explicit unverifiedCompletion rate instead.

Read path (worker/reliability/reader.ts) and CLI (scripts/inspect-capability-reliability.ts → npm run protocol:inspect-capability-reliability): read-only cohort lookup with drift detection against the linked execution_outcomes row. No dashboard, no HTTP route in this slice — that's #191.

CI: extends the closed application-ACL inventory so the ordinary app role gets exactly SELECT, INSERT on both new tables (no UPDATE, no DELETE), and updates the four pinned migration-count/timestamp literals for migration 0031 (31→32 migrations, new max timestamp).

One non-additive change: upsertExecutionOutcome now returns the stored row id so ingest can link to it. Existing callers that ignored the previous void return are unaffected.

Acceptance Criteria Validation

Full mapping in §14 of the design doc. Two are explicitly partial rather than claimed complete:

• Two attempts with materially different capability scopes are not silently combined — cohort fingerprint over project/capability/scope/runtime/policy
• Runtime/model or policy-version changes are visible and can trigger requalification — component fingerprints stored alongside the cohort; a change produces a new cohort automatically
• Every ledger entry links to a canonical outcome and verification/evidence state — execution_outcome_id NOT NULL
• Reprocessing an attempt is idempotent — unique index + ON CONFLICT DO NOTHING, proven in both the mocked and gated-postgres suites
• Metrics can be recomputed deterministically from stored attempts — pure function, no cache
• A critical failure remains visible regardless of the aggregate pass percentage
• Missing independent verification is not counted as a verified pass — self_reported/human_review never count toward independentlyVerifiedPass
• [~] Human rejection, rollback, and override events affect the reliability view — human decisions are fully wired from decideReviewGate; rollback/override have storage, metrics, and tests but no producer until [FEATURE] Add evidence-based earned autonomy policy engine #189/[FEATURE] Add Project Sentinel detection and escalation flow #190 exist to emit them
• [~] "historical data" — covered as attempts predating the table being readable as missing evidence, not as backfilled rows (consistent with ADR 0010)

Tests / Verification

• npm run test:unit:zero-skip — full suite green except one pre-existing, unrelated flake (epic-172-s6-runner-failure-injection.test.ts, a process-kill/deadline test that fails identically on main before this change — confirmed via git stash).
• New suites: capability-reliability-contracts.test.ts, capability-reliability-metrics.test.ts (20 tests, pure-function/fingerprint coverage), capability-reliability-ledger.test.ts (mocked-DB ingest: idempotency target, independent_agent refusal, flag-off, missing/overflow classification, multiplicity), capability-reliability-schema.test.ts (migration text assertions + CI ACL gate assertion), capability-reliability-ledger.postgres.test.ts (gated proof of append-only triggers, gapless sequencing, duplicate-insert rejection, and role privileges — mirrors operation-ledger.postgres.test.ts, requires FORGE_RELIABILITY_LEDGER_REQUIRE_POSTGRES_TEST=1 plus DATABASE_URL/admin URL to run; skips cleanly without them).
• Updated review-gates.test.ts's pinned dbSelect call count (6→8) to account for the two new best-effort adjudication lookups.
• npx tsc --noEmit and npx eslint . both clean.
• No UI change, so no browser verification was applicable.

Risks / Follow-up

• Review requirement is both, with Security/Adversarial review applying per AGENTS.md — this slice touches durable evidence, an append-only audit boundary, and database privileges. Worth particular attention: the ACL inventory edit (.github/workflows/web-ci.yml) and the no-free-text invariant across both new tables.
• Rollback/override adjudication producers land with [FEATURE] Add evidence-based earned autonomy policy engine #189/[FEATURE] Add Project Sentinel detection and escalation flow #190.
• Independent-agent verification stays refused until [FEATURE] Add independent Verification Workforce execution #188 lands.
• The receipt-envelope idea from the community comment on [FEATURE] Add capability reliability ledger #186 remains deferred per §16 of the design doc — a product decision for @Joncallim, not absorbed into this slice.